Method and apparatus for background replacement in still photographs

ABSTRACT

A first digital image is acquired of a framed area while illuminating the background and foreground object under a first lighting condition. A second digital image is then acquired of the same framed area while illuminating the background and foreground object under a second lighting condition. Preferably, the first lighting condition illuminates the background without illuminating the foreground object so that a silhouette of the foreground object is acquired in the first image. The second lighting condition illuminates the foreground object (e.g., with frontal lights). Due to the difference in the illumination between the background and silhouette in the first image, an alpha mask can be created from the first acquired image. Using the mask, the background from the second image can be removed and replaced by virtually any other desired background image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/897,305, filed on Oct. 4, 2010, and entitled METHOD AND APPARATUS FORBACKGROUND REPLACEMENT IN STILL PHOTOGRAPHY, which is a continuation ofU.S. Pat. No. 7,834,894, filed on Apr. 3, 2007, issued on Nov. 16, 2010,and entitled METHOD AND APPARATUS FOR BACKGROUND REPLACEMENT IN STILLPHOTOGRAPHY, the disclosures of which are hereby incorporated byreference in their entireties.

TECHNICAL FIELD

This invention relates generally to the field of photography; moreparticularly, to portrait and still life photography; and moreparticularly still to determining, removing and replacing the backgroundin a portrait or still life photograph with a new background therebycreating a different composite image.

BACKGROUND

It is often desirable in photography to separate the background of animage from the foreground object. For example, photographic studios,professional photographers, and others performing commercial portraitwork (collectively referred to herein as “photographers”) often takepictures of humans and/or still life objects and are asked to deliverimages of the subjects without the original background and/or with adifferent background. The request may be for advertising purposes,uniformity of a plurality of images (e.g., for yearbook oridentification card pictures), and the like.

In the past, several methods have been employed to remove the backgroundfrom the foreground subject in an image. A first prior method utilizesvarious tools in software packages such as Adobe Photoshop (by Adobe)and Paint Shop Pro (by Corel). These tools are often very laborintensive and generally comprise using erase functions and lasso styletools. In the former, the edge of the foreground object is located byhand, and the background is erased using the tool. In the latter, a toolfinds edges automatically when the object is selected. However, the toolis often imprecise due to contrast and color issues between theforeground object and background. Therefore, these tools sufferdrawbacks being labor intensive and imprecise, especially in cases wherethe foreground and background colors cannot be selected in advance. Atleast one prior art reference has termed these types of systems as bruteforce techniques.

Another method of determining the background using these types ofsoftware packages can be performed manually by splitting the image intothe RGB channels. The image with the highest contrast between theforeground object and background can be selected, and then a mask can becreated manually through a series of steps. The mask is used togetherwith the original image to eliminate the background. However, thisprocess suffers a drawback in that it is not automated and depends on amanual selection of a highest contrast channel and creation of the mask.

Chroma key replacement for background removal has also been performed byutilizing a monochromatic background. Typically green screens are usedwith human subjects in order to provide a contrast with skin colors. Anautomated system strips away portions of the image pixel by pixel byassessing whether the pixel has a color that lies within a preprogrammedrange of colors. Several disadvantages associated with this systeminclude inadvertently stripping away any pixels (in this example green)that are located on the foreground object of interest, stripping ofborder portions of the foreground object due to reflectance of thebackground green color onto the foreground object, and stripping ofreflective objects located on the foreground object (e.g., watches,jewelry, etc.). Accordingly, chroma key replacement has a number ofdrawbacks when used in connection with fine photography, and especiallyin those instances where colors of the foreground object are notcontrolled and/or known in advance.

Another example of a prior system employed to eliminate backgrounds isshown in U.S. Pat. No. 6,885,767 to Howell. This system intentionallycreates a background which is much brighter than the foreground object.In this manner, the differential in brightness between the foregroundand background is used to discriminate between the two. The systemutilizes a background with a very high degree of reflection, whereinincident light tends to reflect back along the path from which it came.A device, mounted on the front of the camera, includes a strobe, apartially silvered mirror, and a condenser lens. When the shutter opens,the light source causes intense light to impinge on the object and thereflective background. Due to the high reflectivity, the background isbrighter than the foreground object. This image is then used as a maskfor other shots of the object where the strobe is not triggered.Accordingly, this process uses an intensity technique, rather than achroma technique to eliminate the background. However, this method alsohas several drawbacks, including the requirement of a particularreflective background and special equipment to create the desired lightintensity coming from the plane of the camera lens. The method also hasdrawbacks if taking photographs of humans. More particularly, while themethod may be suitable for photographing still life objects, if theobject moves between the images, then the mask will not registerproperly with the image of the object in the other images.

U.S. Pat. No. 5,574,511 to Yang et al. illustrates yet another method ofutilizing light intensity to create a mask. Here, two IR images withdifferent intensities of IR illumination in the foreground andbackground are compared. A mask is then created which is applied to avisible light image. Here again, the method has drawbacks includingrequiring special equipment, such as IR illumination sources and asecond camera having an IR pass filter. Further, in the case ofphotographing humans, movement of the subject between the shots maycreate a rough match for the mask. Also, the IR mask is not preciseenough for fine photography.

Still another system is disclosed in published U.S. Patent Application2003/0035061. In this system, a still life object is set on a turntablerotating at a constant velocity. A series of images are taken, in analternating manner, with the foreground object first illuminated andthen the background illuminated in a manner to create a silhouette ofthe foreground object. In one of the embodiments described in thepublication, a background cutout unit is disclosed for combining aforeground image picture with a silhouette image to deliver a cutoutimage. In the cutout image, only the foreground images still appear.Accordingly, the cutout image can be combined at a later time with otherbackground images. There are several drawbacks to the system disclosedin this publication. First, the system is employed with still life 3-Dobjects. Therefore, it does not take into consideration photographingobjects which may move in a direction and/or manner other than the fixedrotational velocity of the turntable. Accordingly, photographing humanswho may move between images is not considered. Second, the cutout maskis created with the object area cut-out. The mask is then used in areversed mask layer subtraction process to remove the background. Also,the original image is not described as being preserved andtransmitted—even though this image (and its attendant metadata) may bedesired and/or used in other downstream processing.

Therefore, there is a need in the art for a method, apparatus and systemwhich facilitates taking images of foreground objects in a manner inwhich the background can be determined, removed and replaced withoutrelying on manual methods, chroma replacement, and/or other special IRcameras or equipment mounted in front of the camera to illuminate theforeground object. The invention should also overcome the drawbacksassociated with foreground objects which may move and should not requirespecial backgrounds or predetermined colors of the foreground object.Aspects of the present invention overcome these and other shortcomingsof the prior art and address these needs in the art.

SUMMARY

The invention relates to a method, apparatus and system for selectivelyseparating out a foreground object of interest from a still photograph.In a preferred embodiment constructed in accordance with the principlesof the present invention, two digital images are collected. One imageincludes lighting that illuminates both the foreground object and thebackground. The other image is collected with only the backgroundilluminated, thereby creating a silhouette of the foreground object.This latter image is used to create an accurate mask which, whenregistered with the image with the foreground lighting, allows for theremoval of the background.

One environment in which the present invention may be employed is inconnection with portrait photography. For convenience, this environmentwill be used in describing the embodiments set forth herein. However, itshould be appreciated that other types of still photography may employthe principles of the present invention. For example, still lifephotography and other photographs used in connection with advertising ofproducts, food, etc. (e.g., instances where it may be desirable and/ornecessary to remove or replace the background) are other representativeenvironments.

More specifically, the present invention provides a method, apparatusand system for isolating a substantially stationary foreground object ofinterest included in a captured digital image from the background in thecaptured digital image. Preferably, a first digital image is acquired ofa framed area while illuminating the background and foreground objectunder a first lighting condition. A second digital image is thenacquired of the same framed area while illuminating the background andforeground object under a second lighting condition. Preferably, thefirst lighting condition illuminates the background without illuminatingthe foreground object so that a silhouette of the foreground object isacquired in the first image. The second lighting condition illuminatesthe foreground object (e.g., with frontal lights). Due to the differencein the illumination between the background and silhouette in the firstimage, a mask can be created from the first acquired image. Using themask, the background from the second image can be removed and replacedby virtually any other desired background image.

The present invention preferably accounts for contributions by theforeground object and the background to the intensity level of eachpixel in the border region between the two. A mixing function or alphamask having values between 0 and 1 at each pixel may be employed. Noneof the above prior-art references discloses or suggests taking intoaccount this transition between the foreground object and background.

One feature of the present invention is that the background (e.g., aback drop which comprises the background) in the captured images may bevirtually any monochromatic color or constructed of anysubstrate/materials (e.g., fabric, vinyl, etc.). Because the principlesof the present invention utilize the illumination contrast between thetwo images, the background does not have to be any particular colorand/or constructed of a special material. Therefore, the photographerdoes not have to coordinate colors in advance and/or carry a largeselection of backgrounds.

Another feature of the present invention is that the foreground objectcan be combined into a composite shot with any number of backgroundswhich are preexisting and/or taken at a later time. For example, a groupof students may be photographed in front of an original background atthe start of a school year. During the year, the foreground objects(e.g., the images of the students) may be removed from the originalbackground and placed in front of a second background using the schoolcolors and mascots for school identification cards. The photographs mayalso be used with the original background or with a different backgroundfor a yearbook. Later that year, individual photographs of their childmay be delivered to parents, where the parents can pick differentpredetermined backgrounds for combination with the photograph of theirchild. While the predetermined backgrounds can be of virtually anylocation or background, representative backgrounds might include theschool library, stadium, quad, or other location on campus.

Still another feature of the present invention is that the imageacquisition sequence may be selected to reduce the elapsed time betweenthe acquired images. In this case, the silhouette image is capturedfirst and the normally illuminated foreground image is captured second.In this manner, the lighting sequence can be optimized to reduce theamount of time between the two images. For example, it will beappreciated the photographic flash lighting generally includes a fairlysteep front edge and then decays. Accordingly, if the photograph of theforeground object was taken first, a relatively long delay would benecessary while waiting for the front flash illumination to decay (i.e.,if the second image was acquired too quickly, the foreground objectwould still be illuminated and a good silhouette image would not becaptured). The present invention, however, preferably takes advantage ofthe background flash profile by acquiring the silhouette image first,and then sequences the acquisition of the second image to theappropriate time during the decaying background light.

This manner and sequence of image acquisition reduces the time betweenacquisition of the images and reduces the possibility of movement of theforeground object in the interval between the capture of the images. Itwill be understood that movement of the foreground object between imagesreduces the registration of the mask of the foreground object (createdfrom the captured silhouette image) and the normally lit image of theforeground object. Therefore, this feature is very useful whenphotographing young children, pets, and other objects which tend to moveduring photographs. Other movement of the foreground object, such asnormal involuntary movement of a person being photographed, is permittedby taking the backlit and front-lit images within a minimized timeinterval. Alternatively, if the foreground object cannot move and/or isnot likely to move in the time period between image captures, then thesequence of image captures may be reversed. In such an event, the timeperiod between such image captures may be increased to virtually anyarbitrary time period.

Therefore, according to a first aspect of the invention, there isprovided a method of imaging a foreground object placed in front of abackground, the method comprising: acquiring a first image of theforeground object and the background while illuminating the backgroundunder a first lighting condition relative to the object; acquiring asecond image of the foreground object and the background whileilluminating the background under a second lighting condition relativeto the object; computing a mixing function based on a selected one ofthe images; and using the image that was not selected and the mixingfunction, computing an object image function relating to an image of theforeground object with a reduced background intensity relative to theforeground object as compared to the image that was not selected.

According to the aspect recited in the preceding paragraph, there may beprovided the selected image being the first image and the step ofcomputing a mixing function includes computing an alpha value for alphablending for each pixel of the first image. Further, the first image maybe acquired under the first lighting condition, which conditioncomprises illuminating the background so that it is brighter than theforeground object.

According to a second aspect of the invention, there is provided amethod of computing an image, the method comprising: computing a mixingfunction from a first image taken of a scene having a foreground objectplaced in front of a background, where the background is illuminatedunder a first lighting condition relative to the object; accessing asecond image of the foreground object and the background, where thebackground is illuminated under a second lighting condition relative tothe object; and using the second image and the mixing function,computing an object image function relating to an image of theforeground object with a reduced background intensity relative to theforeground object as compared to the second image.

According to a third aspect of the invention, there is provided a methodof imaging an object placed in front of a background, the methodcomprising: acquiring a first image of the foreground object and thebackground while illuminating the background under a first lightingcondition relative to the object; acquiring a second image of theforeground object and the background while illuminating the backgroundunder a second lighting condition relative to the object within apredetermined time interval of acquiring the first image, thepredetermined time interval being computed based on a maximum acceptablespeed of movement of the foreground object.

According to a fourth aspect of the invention, there is provided animaging system for imaging a scene having a foreground object placed infront of the background, comprising: a camera system adapted to acquirean image of the scene; and a lighting system configured to generate afirst lighting condition, in which the background appears to the camerabrighter than the foreground object, and a second lighting condition, inwhich both the foreground object and the background are illuminated,wherein the lighting system is arranged and configured to sequentiallygenerate the first and second lighting conditions within a time intervalcomputed based on a maximum acceptable movement speed of the foregroundobject, and the camera system is adapted to acquire a first image of thescene under the first lighting condition and a second image of the sceneunder the second condition.

According to a fifth aspect of the invention, there is provided astorage medium having encoded thereon computer-readable instructionsthat, when executed by a computer, causes the computer to: compute amixing function from a first image taken of a scene having a foregroundobject placed in front of a background, where the background isilluminated under a first lighting condition relative to the object;access a second image of the foreground object and the background, wherethe background is illuminated under a second lighting condition relativeto the object; and using the second image and the mixing function,compute an object image function relating to an image of the foregroundobject with a reduced background intensity relative to the foregroundobject as compared to the second image.

According to a sixth aspect of the invention there is provided a methodfor isolating a substantially stationary foreground object in a framedarea from the background in the framed area using still photography,comprising: acquiring a first image of the framed area, whileilluminating the background and foreground object under a first lightingcondition; acquiring a second image of the framed area whileilluminating the background and foreground object under a secondlighting condition; determining the background from a predetermined oneof the first and second acquired images; and removing the backgroundfrom the other image by registering the determined background of thepredetermined one image with the other image.

According to the aspect recited in the preceding paragraph, the firstlighting condition may illuminate the background without illuminatingthe foreground object so that a silhouette of the foreground object isacquired in the first image; the predetermined image may be the firstimage and the other image is the second image; the second lightingcondition may illuminate the foreground object with frontal lights; andthe step of determining the background may include determining an alphamask.

According to a seventh aspect of the invention, there is provided asystem for isolating a substantially stationary foreground object in aframed area from the background in the framed area using stillphotography, comprising: a front light and a back light arranged andconfigured to generate a first lighting condition, in which thebackground appears brighter than the foreground object, and a secondlighting condition, in which both the foreground object and thebackground are illuminated; a camera system adapted to acquire a firstand second image of the framed area during the first and second lightingcondition, respectively; and a processor, operatively connected to thecamera system, the processor including processing means for determiningan alpha mask from a predetermined one of the first and second images,the alpha mask being used to remove the background from the other of thefirst and second images.

The aspects are numbered in the preceding paragraphs for convenience,and not by way of limitation. Further, while the invention will bedescribed with respect to preferred embodiment configurations, and withrespect to preferred foreground objects and image acquisition sequence,it will be understood that the invention is not to be construed aslimited in any manner by either such configuration, foreground objectsor image acquisition sequence described herein. Instead, the principlesof this invention extend to any environment in which two digital imagesare taken sequentially where one of the images includes a silhouette ofa foreground object of interest. These and other variations of theinvention will become apparent to those skilled in the art upon a moredetailed description of the invention.

The advantages and features which characterize the invention are pointedout with particularity in the claims annexed hereto and forming a parthereof. For a better understanding of the invention, however, referenceshould be had to the drawings which form a part hereof and to theaccompanying descriptive matter, in which there is illustrated anddescribed a preferred embodiment of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings in which like elements are identified with the samedesignation numeral:

FIG. 1 a illustrates a digital photographic image in which both theforeground object and the original background are illuminated.

FIG. 1 b illustrates a digital photographic image in which theforeground object is backlit, thereby creating a silhouette of theforeground object.

FIG. 1 c illustrates a calculated a mask made from the image of FIG. 1b.

FIG. 1 d illustrates a digital photograph of a selected backgroundimage.

FIG. 1 e illustrates a composite photograph of the foreground object ofFIG. 1 a in which the α mask of FIG. 1 c is used to eliminate theoriginal background and replace it with the selected background of FIG.1 d.

FIG. 2 schematically shows an embodiment of a system according to oneaspect the invention for acquiring the normally lit and silhouetteimages.

FIG. 3 schematically shows an interline-transfer CCD sensor used in asystem according to one aspect of the present disclosure.

FIG. 4 diagrammatically illustrates the timing of the image capture bythe camera 210 relative to the illumination from the light sources andthe camera 210 integration.

FIG. 5 a is a first alternative embodiment to camera 210 in which twodigital image sensors with electronic shutters are employed, and whereina device for sending the same image to the two sensors is employed.

FIG. 5 b is a second alternative embodiment to camera 210 in which twodigital image sensors are employed with mechanical shutters, and whereina device for sending the same image to the two sensors is employed.

FIG. 5 c is a third alternative embodiment to camera 210 in which twodigital image sensors are employed, wherein each digital image sensorincludes a shutter and lens, and a device for sending the same image tothe two sensors is employed.

FIG. 6 schematically shows an enlarged portion of the image in FIG. 1 cin a region around a segment of an edge between the foreground objectand background in which the transition between light and dark pixels isshown.

FIGS. 7( a) and 7(b) show, in different scales, a curve representing therelationship between the maximum tolerable speed of motion of theforeground object and the maximum lapsed time allowed between capturingthe silhouette image and front-lit image according an aspect of thepresent disclosure.

FIG. 8 schematically illustrates a larger distributed system 800 inwhich images captured by the system 200 of FIG. 2 may be processed,downloaded, and stored, among other further uses.

DETAILED DESCRIPTION

The invention relates to methods, apparatus and systems for selectivelyidentifying and removing the background from a digital photographicimage which includes a background and a foreground object. A replacementbackground may then be combined with the foreground object to create acomposite photograph. In particular, the invention relates to methods,apparatus and systems for sequencing two photographs, creating and usinga mask for removing the original background, and creating new compositephotographs.

Generally, the imaging process includes capturing a backlit image of theforeground object. The resulting backlit, or silhouette, image is usedto determine a mask. A normal, front-lit image of the object, i.e., animage with both the object and background illuminated is also captured.The front-lit image and the mask are combined to create an image of theobject with the background removed. As previously discussed, the imagesmay be sequenced in certain manners in order to take advantage oflighting characteristics and other considerations. In a preferredembodiment, the silhouette image is taken first and the front-lit imageis taken second. However, the principles of the present invention arenot limited to that order. Accordingly, while the terms first image andsecond image are used herein, it should be noted that the terms firstand second are used to differentiate between the two images and are notmeant to imply a temporal acquisition—unless the context specificallyindicates otherwise.

Referring now to FIGS. 1 a-1 e, there is illustrated a series of imageswhich depict an overview of the present invention. In FIG. 1 a, thedigital photographic image 20 includes a foreground object 21 (in thiscase a portrait of a person) and a background 22. In FIG. 1 a, both theforeground object and the original background are illuminated in anormal manner. The framed area of the image is designated at 23. FIG. 1b illustrates a digital photographic image 20′ acquired from the samecamera location such that the framed area of the image remains unchangedand is designated 23. However, since the foreground object 21 wasbacklit, a silhouette of the foreground object 21 is created. FIG. 1 cillustrates a calculated α mask 24 made from the image of FIG. 1 b(described more fully below). FIG. 1 d illustrates a representativedigital photograph of a selected background image. Finally, FIG. 1 eillustrates a composite photograph of the foreground object 21 in whichthe α mask 24 is used to eliminate the original background 22 andreplace it with the selected background of FIG. 1 d.

Turning now to FIG. 2, a system 200 is illustrated which may be utilizedin connection with the acquisition of the sequential images. A digitalcamera 210 is used to acquire images of a foreground object or person205 located in front of a background 215. While preferably thebackground 215 is selected to provide a useful, high quality photographof the captured front-lit photo, the background 215 can include a backdrop of virtually any monochrome color and made of any material. Thecamera 210 is connected to a controller 240 which provides thesynchronization of lights 220, 225, and 230, as well as the imagecapture by camera 210.

The controller 240 operates to initiate, in a predetermined sequence,the lights 220 and 225 and/or 230 and the shutter of the camera 210. Thecontroller can be of a variety of types, including a digitallyprogrammable device, a circuit responsive to a synchronization triggeroutput from the camera 210 to sequence the lights 220, 225, and 230, ora computer-operated controller.

The system 200 further includes a processor 260, which can beoperatively connected to the controller 240, camera 210, and/or lights220, 225, and 230. The processor 260 preferably includes a centralprocessing unit 261 for running necessary application and systemprograms and a memory location 262. In addition, processor 260 ispreferably arranged and configured to provide a memory storage device263 for receiving and storing image data from camera 210. The memorystorage device 263 may be an external hard drive connected with a USBtype cable, a floppy disk drive, a CD or DVD writer or other well knownmemory storage device. The processor 260 may be a stand-alone device,wherein the image data from the camera 210 is provided via flash drive,memory stick, floppy disk, CD, DVD, or other well known data storagedevice. The processor 260 can be of a variety of types, including one ormore computers, including special purpose computers or general-purposecomputers such as personal computers (e.g., a Pentium chip based PC).The processor 260 is preferably programmed to implement at least partsof the image computing algorithms (described further below) inconnection with the sequential acquired images. However, the processor260 may merely record the sequential acquired images and transfer theimages to a remote or other computer (best seen in FIG. 8).

While not specifically shown in FIG. 2, processor 260 may generallyinclude various PC components and devices such as a video display unit,various memory devices (e.g., hard drives, CD-Drives, etc.), user inputdevices (e.g., a mouse and/or keypad), network connections forconnecting to the internet and providing communications capability, anda modem.

The illumination units or lights 220, 225 and 230 are preferably flashunits such as those sold under the designation AlienBees manufactured byAlienBees, a Division of Paul C. Buff, Inc. of Nashville Tenn. for thefront light 220 and such as those sold under the designation Lumedynemanufactured by Lumedyne Inc. of Port Richey, Fla. for background light225 and rear light 230. While not specifically shown in FIG. 2, itshould be understood that a plurality of lights may be used rather thana single light—and that a single light for illuminating each of theforeground, background and translucent back drop is shown in FIG. 2merely for convenience. While other types of lights and illuminationdevices may be used, an important selection criteria for the lights isthe ability to backlight the foreground object in a manner which createsa silhouette and to light the foreground object in a manner whichpermits the capturing of an image of appropriate and/or desired quality.

The rear light 230 may optionally be used in connection with atranslucent background 215. This type of lighting may provide more evenlighting in the captured image and with fewer shadows, as well as theability to locate the person 205 nearer the background screen 215. In apreferred embodiment only one of the lights 225 or 230 are generallyused to capture the silhouette image. However, the lights 225 and 230may be used together if desired.

The rear lighting may also be optionally generated using a back dropcoated with a light emitting, glowing or other luminous layer 217. Suchlight can be obtained from a variety of luminescent materials. Forexample, chemiluminescence and bioluminescence can be used. These lightsources involve the emission of light from chemical or biochemicalreactions at ordinary temperatures. Such sources are described in moredetail further below.

In the preferred embodiment, the camera 210 is comprised of three mainsections: the lens 213, a mechanical shutter 212, and a CCD element 211.Generally, CCD elements have relatively rapid exposure speeds. However,the process of moving the captured image from the CCD element 211 to animage storage area is slower than the time to acquire the image.Accordingly, in order to reduce the time between acquiring the backlitand front-lit images—preferably to further reduce any motion of theforeground object in the time period between shots—the preferred CCDelement 211 is an interline transfer CCD. Such elements are commerciallyavailable, and are manufactured by Eastman Kodak Company of Rochester,N.Y. under the designation KAI-11000. This type of CCD includes arraysof photodiodes interspaced with arrays of shift registers (best seen inFIG. 3 at 1000). In operation, after capturing a first image, thephotodiodes 1010 transfer the electrons to the adjacent shift registersand become ready thereafter to capture the next image. Because of theclose proximity between the photodiodes and associated shift registers,the imaging-transfer cycles can be very short. Thus, the preferreddevice can rapidly capture a first image, transfer the first image to amemory location (where it is temporarily stored) and then capture asecond image. After the sequence of images, both of the images can bedownloaded to the appropriate longer term memory location.

Since the CCD element 211 continues to integrate the second image whilethe first image is read out, a shutter 212 is employed in front of theCCD element 211. In the preferred embodiment, a mechanical shutter 212is used and is synchronized by controller 240. The shutter 212 opensprior to the capture of the first image and remains open for theduration of the second flash. It then receives a signal to close inorder to eliminate further exposure from ambient light. The preferredshutter 212 is commercially available, such as those manufactured byRedlake MASD LLC of Tucson, Ariz. However, other shutters may beemployed. Further, the exposure may be controlled by the strobes,shutter, and/or a combination of the two.

Lens 213 is located in front of shutter 212 and is selected to providethe appropriate photographic characteristics of light transmission,depth of focus, etc. In the preferred embodiment, lens 213 is selectedbetween 50 and 250 mm, with the image taken at an f-stop generally inthe range of f16 to f22. This provides a zone focus for the image. Italso generally eliminates concerns regarding ambient light. However, itwill be appreciated that any number of lenses, focusing, and f-stops maybe employed in connection with the present invention.

Camera 210 is arranged and configured to provide a single trigger pulseat the start of the integration of the first image. This pulse may beused by the controller to synchronize the lights 220, 225, and 230. Inone embodiment, the front or rising edge can trigger the backgroundlights 225 and/or 230, while the trailing or falling edge can triggerthe front light 220. Other types of triggers and pulses may be used. Forexample, camera 210 might use two different pulses, etc.

To initiate the capture of the images, a shutter release (not shown) ispreferably used. Such a release is generally connected to the camera.However, other methods and devices may be used to initiate the imagecapture. For example, the button, switch or other device might beincluded on the controller 240. Still further, the computer 260 could beused to initiate the process.

FIG. 4 illustrates the preferred timing for the actuation of the variousdevices of system 200. As previously discussed, the strobe lights arethe pacing items for the acquisition of the two sequential images due tothe decay of the light from the strobes. Certain types of lights may beemployed to narrow the time period. In a preferred embodiment, the syncpulse is generated by camera 210. The controller 240 synchronizes thestrobes, wherein strobe 1 correlates with the activation of backgroundlight 225 and strobe 2 correlates with the activation of front light220. As noted above, rear light 230 may be optionally employed with atranslucent background 215 either in combination with background light225 or by itself.

The timing of the activation of strobe 2 and the capture of the secondimage (designated Exposure 2 in FIG. 4) is preferably selected so thatthe decay of the background light 225 reaches a desired point foracquiring the image of the foreground object. Therefore, a singleactivation of the background strobe may be employed for both of thecaptured images. Exposure 2 is started electronically by the camera 210,but is finished by the shutting of the mechanical shutter 212. As shownin FIG. 4, it takes approximately 10 milliseconds to close theshutter—with most of the time lag being due to getting enough current inthe solenoid when driven with a constant voltage. However, such timeperiod may vary using other types and styles of shutters and drivecircuits.

The Integration 1 and Integration 2 time periods (abbreviated Integratein FIG. 4) are those periods in which the light energy falling on theCCD element 211 are converted to an electronic light intensity.Therefore, the time period t₁ generally corresponds to the Exposure 1and time period t₂ generally corresponds to the Exposure 2. In thelatter case, the Integration 2 continues after the mechanical shutter212 closes, but since the closed shutter blocks all further incidentlight, the Exposure 2 is not affected. There is a very short delaybetween the end of the Exposure 1 and the start of the Exposure 2 whichis due to moving the collected image information in the preferred CCDelement 211 to its on-board memory location proximate the photodiodecollectors.

Alternative cameras and other image acquisition devices may be employedto practice the principles of the present invention. For example, FIGS.5 a-5 c illustrate three different embodiments which may be utilized toacquire the sequential images.

In FIG. 5 a the alternative embodiment is shown generally at 500. TwoCCD devices or other digital image sensors 501, 502 are utilized. Eachof the digital image sensors include integral electronic shutters (notshown). A beam splitter 503 is used to provide the same image from lens504 to the digital image sensors 501, 502. In this embodiment, the firstand second images may be captured by sensors 501 and 502, respectively.

A second alternative embodiment is shown in FIG. 5 b at 510. Two CCDdevices or other digital image sensors 511, 512 are utilized. However,each of the sensors 511, 502 in this embodiment are used in connectionwith mechanical shutters 513, 514 respectively. A beam splitter 515 isused to provide the same image from lens 516 to the digital imagesensors 511, 512. In this embodiment, the first and second images arecaptured by sensors 511 and 512, respectively.

A third alternative embodiment is shown in FIG. 5 c generally at 520.Two CCD devices or other digital imaging sensors 521, 522 are utilized.Each of the sensors 521, 522 are used in connection with a mechanicalshutter 523, 524 respectively. Also, an individual lens 525, 526 isprovided with each sensor. A beam splitter 527 provides the image toeach of the sensors 521 and 522 via lenses 525 and 526. In thisembodiment, the sequential images are captured by sensors 521 and 522.Preferably, the related sensor 521, shutter 523, and lens 525 (andsensor 522, shutter 524, and lens 526) may be combined in an integralmanner—essentially providing two cameras taking a picture of the sameimage with a beam splitter. However, discrete components may also beemployed.

In each of the three alternative embodiments, other devices may beemployed in lieu of beam splitter 503, 515, and 527 to provide thedesired functionality. For example, prisms, beam splitter cubes, mirrorswith holes, and other devices which send the same image to two physicallocations may be used. Further, while additional optical devices andsensors are required in the three alternative embodiments, theembodiments may have advantages in speed and downloading of the imagefrom the sensors. However, the lights may still be the pacing item forthe necessary time for the acquired images.

Turning now to FIG. 6, a representative area of the boundary between theremoved background and the mask portions of FIG. 1 c is schematicallyshown. The boundary is shown generally at 600. In order tosatisfactorily separate the foreground and background portions, it isnecessary to consider the details of the boundary between the twoportions. Due to the resolution of the optics and number of pixels ofthe CCD device 211, there is not a sharp line of demarcation between thebackground 22 and the silhouette 21. Instead, as shown in FIG. 6, thereis a blurring over a number of pixels. More specifically, there is atransition from α=1 in the silhouette or mask area, proceeding through arange of pixels where 0<α.<1 in the blurred area, and reaching thebackground area where α=0. Thus, in this border region, the pixelsinclude contributions from both the foreground and background portions.The present invention takes the fractional contributions from bothforeground and background into account when removing the originalbackground and combining new backgrounds.

In order to remove the original background, a mixing function is used,which in the preferred embodiment uses the alpha channel, whichcomprises an alpha value (α) for each pixel. First, an estimate of thebacklit background B^(bl) at each point in the image is made. Then, foreach point in the image, using the brightest color channel (i.e., theblue channel if a blue background is used or the green channel if agreen or gray background is used), β is calculated as follows:

β=M _(g) /B _(g) ^(bl)  (1)

-   -   where β is the ratio of background to foreground    -   M_(g) is the measured pixel level    -   B_(g) ^(bl) is the estimated background pixel level

Next, α is calculated as follows:

α=1−(β−β_(l))/(β_(h)−β_(l)) if β_(l)≦β≦β_(h)

α=1 if β<β_(l)

α=0 if β>β_(h)  (2)

-   -   where α is the mixing factor    -   β_(l) is low β threshold, below which α=1    -   β_(h) is high β threshold, above which α=0

In the preferred embodiment, β_(l) is set such that in the knownforeground α=1 with the observed noise and ambient light contamination.β_(h) is set such that in the known background α=0 with the observedimage noise.

A variety of estimation methods can be used. For example, it may bepossible to create a uniform background. In such cases B_(g) ^(bl) isthe same for every pixel. For non-uniform background, surface fittingtechniques may be used.

As discussed above in connection with FIG. 6, in border regions betweenthe foreground and background portions of an image, α takes on valuesbetween 0 and 1. Thus, α undergoes a transition between 0 and 1,reflecting the partial contributions of the intensity level in thepixels in the border regions. The resulting alpha channel results of theimage comprise the mask which is used as described below to remove theoriginal background from the image in which the foreground object iswell-lit.

To remove the background, an object image function is computed. Theobject image function in this example is αF^(c) and is computed by theformula

αF ^(c) =M−(1-α)B ^(fl)  (3)

-   -   where F^(c) is the corrected foreground pixel (i.e., without any        background mixed in),    -   M is the measured foreground image pixel,    -   B^(fl) is the estimated front lit background pixel.        As with the backlit image, the background here is estimated,        typically from the light intensities deep with background image        portion 22. For cases where the background is sufficiently        uniform, B^(fl) is a constant for all pixels; for a non-uniform        background, well-known techniques can be used to model the        background. Note that this formula is a rearrangement of the        formula for conventional alpha blending, according to which

R=αF+(1−α)B  (4)

-   -   where R, F, and B are color vectors with red, green and blue        components,    -   α is the mixing factor,    -   F is the foreground pixel,    -   B is the background pixel, and    -   R is the pixel resulting from alpha blending of the foreground        and background pixels.

Because α is known and αF^(c) is known, F^(c) can be computed.Alternatively, as shown below, because αF^(c), instead of F^(c) itself,is used for subsequent compositing with new background images, α andeither F^(c) or αF^(c) can be stored or transmitted to the desireddestinations (e.g., over computer networks such as the Internet) forlater use in compositing with new background images.

The object image function, αF^(c), and a new background image arecombined to generate an image with the foreground image F^(c) in a newbackground. To generate the new image, the standard alpha blending isagain used:

R=αF ^(c)+(1−α)B ^(new)  (5)

-   -   where R is the resulting pixel, and    -   B^(new) is the new background pixel.

In the new image, α is approximately 1 in the areas in registration withthe image of the foreground object 21, and those areas are thereforeoccupied by the image of the foreground object 21. In contrast, α isapproximately 0 in the areas outside image of the foreground object 21.Therefore, those areas are occupied by the new background image. In theborder regions between the foreground and background images, α isbetween approximately 0 and approximately 1. In intensity level of eachpixel in the border regions is thus a sum of the contributions from boththe foreground image and background image. The foreground object 21therefore appears to be placed in front of the new background in thecomposite image, with properly blended edges between the two imageportions. As a result, the foreground object appears naturally in frontof the background.

Example Applications

One feature of the present invention is that the image of the foregroundobject can be positioned anywhere, and in any orientation, relative tothe new background in a composite image. For example, the foregroundobject image can be positioned by padding values of 1 for α and valuesof (0, 0, 0) for F^(c) or α F^(c) for the appropriate pixels. Thus, forexample, a foreground object image taken in a portrait (i.e., vertical)format can be used to position the image of the object in a newbackground with a landscape (i.e., horizontal) format.

Another feature is that once a backlit image and front-lit image arecaptured and processed as described herein, only α and αF^(c) (or F^(c))values need to be stored for later use. The stored information can belater retrieved to combine with new background images to generatecomposite images. The α and αF^(c) information can also be transmittedover computer networks to be used at destination sites. By transmittingonly this information, speed and bandwidth may be significantlyimproved. More specifically, if the new backgrounds are comprised of alarge number of pixels, then the background may be stored at thedestination site, and only the foreground object needs to betransmitted.

By way of example, and with reference to FIG. 8, various components of alarger, distributed system are shown at 800. In this larger, distributedsystem 800, the images captured by individual systems 200 and stored inmemory storage devices 263 may be provided to a centralized database 804for processing, storage, further distribution and/or other uses. Datastore 808 comprises the memory storage device 263 and provides for thecaptured images to be transmitted by computer 807, via internet 801 andweb server 803, to the centralized database 804. Since a number ofsystems 200 may be employed to capture a large number of images, aplurality of data stores 808 may be used. Further, any number ofindividual memory storage devices 263 may be represented by data store808.

Although internet 801 is illustrated as a preferred communicationmedium, other communication systems may be employed including directlyconnecting memory storage devices 263 to the database 804 via LAN orWAN, proprietary communication connections, dial-up modems over PBXnetworks, etc.

In this system, the stored α and αF^(c) information can be transmittedfrom either the computer 807 or the web server 803 to the receivingcomputer 802. Various backgrounds may then be inserted into the image bythe receiving computer 802 having that information.

Yet another feature is providing images over on-line services. Stillreferring to FIG. 8, here a customer at a remote site 809 can be shownan image of the foreground object and be allowed to choose one or morebackgrounds from a predetermined set of background images to be combinedwith the foreground object image. In this case, the remote user 809 mayview the appropriate images 805 and backgrounds 806 stored in thedatabase 804 via the internet 801. Once the customer has selected thebackground or backgrounds and placed the order, then the foregroundobject image (F^(c) or αF^(c)) can be transmitted to the appropriateprinting facility to be combined with the customer-selected backgroundor backgrounds. In this example, the receiving computer 802 may belocated at the printing facility.

As noted above, the set of backgrounds can be pre-stored at the printingfacility, and the customer-selected backgrounds need not be transmittedwith every order, thereby reducing the transmission overhead of thenetwork. Any number of images 805 and backgrounds 806 may be stored inthe database 804, with images 1-n and backgrounds 1-n shown by way ofillustration. The number of users 809, 810 may be comprised of virtuallyany number of users 1-n. The users may be prompted to enter passwords orother information to view the appropriate stored images 805 andbackgrounds 806. The users may also view hard copies of images and/or bephysically present at a physical location while viewing the images andbackgrounds, rather than logging on to a site and viewing the imagesover internet.

As described above, another application is the use of a single image ofa foreground object with a variety of backgrounds for a variety ofpurposes. For example, in a school or university setting, a portrait ofa student can be taken with the process disclosed above, and the imageof the student without the background can be stored. The stored imagecan later be used with appropriate backgrounds for a variety ofapplications, including photo identification cards, yearbooks,announcements, campus news paper photos and student's personal webpages. Efficient use of organizational resources is thereby achieved.

A further feature of the present invention is the use of a backlittranslucent background. By using this arrangement, the subject may beplaced closer to the background. As shown in FIG. 2, a full length shotmay be taken of the foreground object 205. A short stage 216 or othersuitable flooring can be placed under the object and extend to thebackground 215 to enable full length shots. Other creative images may becaptured by utilizing the present invention with a translucent floor. Inthis case the flash is located below the floor and the camera looks downon the object on the floor.

According to another feature of the present disclosure, the timeinterval between the captured images may be adjusted to compensate forthe anticipated movement of the foreground object. By selecting theappropriate interval, the mask and foreground object image will beadequately registered with one another. For example, to capture imagesof a dancer in motion, it may be anticipated that the speed of themotion to be tolerated is much greater than for a still photo. Asanother example, if an image is captured at a high resolution, but onlyneeds to be displayed as a lower resolution, a misalignment of morepixels may be more acceptable than if the image is to be displayed at ahigh resolution.

For a given amount of tolerance of mis-registration, the relationshipbetween the tolerable speed of motion and the time interval between thesequentially captured images is essentially hyperbolic, governed by theequation

v _(t) =w _(s) d _(t)/(w _(c) t)  (6)

-   -   where v_(t) is the tolerated velocity at the subject,    -   w_(s) is the width of the image at the subject,    -   d_(t) is the distance (in pixels) in the camera, that can be        tolerated,    -   w_(c) is the width (in pixels) of the image in the camera,    -   t is the time to capture both images.

Thus, for example, for w_(s)=36 inches, d_(t)=0.5 pixels and w_(c)=2672,the tolerable speed of movement, v_(t), is a hyperbolic function of theimage capture time t, as shown in FIGS. 7( a) and 7(b). As a morespecific example, for a tolerance of mis-registration of a fraction ofpixels (e.g., 0.5 pixels) and a speed of movement of a few inches persecond, the backlit and front-lit images must be taken with a fewmilliseconds from each other. For such applications, camera systems withcapability for capturing images in rapid successions, as discussedabove, can be used.

Yet another feature is the additional accuracy of face finding after theelimination of the background. Resizing heads to a uniform size foryearbooks and the like, sharpening and processing of images, anddropping subjects into multiple layered photos are all additionalfeatures which may be accomplished in connection with the presentinvention.

Luminescent Materials

It is believed that luminescence may be used as the layer 217 onbackground 215 to generate the backlighting in certain conditions.Luminescence is the process of producing light in excess of thermalradiation following an excitation. A solid material exhibitingluminescence is called a phosphor. Phosphors are usually fine inorganiccompound powders of a high degree of purity and a median particle sizeof 3-15 micrometers, but may be large single crystals, used asscintillators, or glasses or thin films. Phosphors may be excited byhigh energy invisible uv radiation (photoluminescence), x-rays(radioluminescence), high energy electrons (cathodoluminescence), astrong electric field (electroluminescence), or in some cases infraredradiation (up-conversion), chemical reactions (chemiluminescence), oreven stress (triboluminescence). Phosphors usually contain activatorions in addition to the host material. These ions are deliberately addedin the proper proportion during the synthesis. The activators and theirsurround ions form the active optical centers. The optical properties ofa phosphor are measured on relatively thick plaques of the phosphorpowder. An important optical property for the application of thephosphor is its emission spectrum, the variation in the intensity of theemitted light versus wavelength.

Electroluminescence methods are particularly suited to the task ofgenerating rapid pulses of light. This type of luminescence involves aphosphor which generates light directly when an applied electric fieldis applied. When impressed across a phosphor the source is mostdesirable for flat panel displays. There are two ways this can be donewith present materials. The first is to use a light-emitting diode(LED). These are single crystal usually of GaP doped with trace amountsof nitrogen. The second way to directly convert electric energy intolight is with an electroluminescent phosphor. By far the bestelectroluminescent phosphor is ZnS:Mn²⁺.

Other methods of luminescence may be used to produce a strobe light inlayer 217, provided that they are fast and bright enough for the currentapplication. In addition, other technologies to provide a strobe lightin layer 217 may be used including arrays of smaller slash units orlight-emitting diodes (LEDs).

While particular embodiments of the invention have been described withrespect to its application, it will be understood by those skilled inthe art that the invention is not limited by such application orembodiment or the particular components disclosed and described herein.It will be appreciated by those skilled in the art that other componentsthat embody the principles of this invention and other applicationstherefore other than as described herein can be configured within thespirit and intent of this invention. The arrangement described herein isprovided as only one example of an embodiment that incorporates andpractices the principles of this invention. Other modifications andalterations are well within the knowledge of those skilled in the artand are to be included within the broad scope of the appended claims.

1. An imaging system for photographing a subject, the system comprising:a foreground light source arranged and configured to illuminate thesubject; a background light source arranged and configured to illuminatea background; a digital camera configured to capture still photographsof a subject arranged between the digital camera and the background; anda controller configured to: synchronize illumination of the backgroundlight source with a capture of a background illuminated image taken bythe digital camera; and synchronize illumination of the foreground lightsource with a capture of a foreground illuminated image taken by thedigital camera.
 2. The imaging system of claim 1, wherein the controlleris arranged and configured to capture the foreground illuminated imagebefore complete decay of light from the illumination of the backgroundlight source.
 3. The imaging system of claim 1, wherein the digitalcamera is arranged and configured to capture color photographs.
 4. Theimaging system of claim 1, wherein the digital camera is arranged andconfigured to detect visible light from the foreground light source andthe background light source, and wherein the foreground light source andthe background light source are positioned to provide visible light tothe digital camera.
 5. The imaging system of claim 1, wherein thedigital camera comprises a lens, a mechanical shutter, and acharge-coupled device.
 6. The imaging system of claim 5, wherein themechanical shutter is synchronized to close after illumination of theforeground light source and during the digital camera's capture of theforeground illuminated image.
 7. The imaging system of claim 5, whereinthe charge-coupled device is an interline-transfer charge-coupleddevice.
 8. The imaging system of claim 7, wherein the interline-transfercharge-coupled device includes photodiodes and shift registers, whereinthe photodiodes are configured to transfer electrons associated with thebackground illuminated image to the shift registers prior to capturingthe foreground illuminated image using the photodiodes.
 9. The imagingsystem of claim 1, further comprising a memory storage device, whereinthe memory storage device stores the background illuminated image andthe foreground illuminated image.
 10. A method of photographing asubject arranged in front of a background, the method comprising:synchronizing a flash of a background light source to occur whilecapturing a background illuminated image with a digital camera, thedigital camera configured to capture still photographs, and thebackground light source positioned to illuminate the background; andsynchronizing a flash of a foreground light source to occur whilecapturing a foreground illuminated image with the digital camera. 11.The method of claim 10, wherein the flash of the foreground light sourceis timed to occur before complete decay of the flash of the backgroundlight source.
 12. The method of claim 10, wherein the backgroundilluminated image and the foreground illuminated image are color images.13. The method of claim 10, further comprising timing the flash of theforeground light source to occur within a time interval of the flash ofthe background light source, wherein the time interval is computedaccording to the formula:t=w _(s) d _(t)/(w _(c) v _(t)) where v_(t) is a tolerated speed of aforeground object, w_(s) is a width of the foreground illuminated imageat the foreground object, d_(t) is the number of pixels by which thebackground illuminated and the foreground illuminated images arepermitted to be offset from each other, w_(c) is the width (in pixels)of the background illuminated and the foreground illuminated images, andt is the time to capture both of the background illuminated and theforeground illuminated images.
 14. The method of claim 10, furthercomprising timing the flash of the foreground light source to occurwithin a time interval of the flash of the background light source,wherein the time interval is computed based on a number of pixels bywhich the background illuminated and the foreground illuminated imagesare permitted to be offset from each other, wherein the offset is lessthan a fraction of a pixel.
 15. The method of claim 10, wherein thedigital camera comprises a lens, a mechanical shutter, and acharge-coupled device, and further comprising closing the mechanicalshutter during the capturing of the foreground illuminated image withthe digital camera.
 16. An imaging system for photographing a subject,the system comprising: a background; a foreground light source arrangedand configured to illuminate the subject; a background light sourcearranged and configured to illuminate the background; a digital cameraconfigured to capture still photographs of a subject arranged betweenthe digital camera and the background; and a controller configured to:synchronize illumination of the foreground light source with a captureof a foreground illuminated image with the digital camera; andsynchronize illumination of the background light source with a captureof a background illuminated image with the digital camera.
 17. Theimaging system of claim 16, wherein the controller generates a singletrigger pulse, wherein a rising edge of the single trigger pulsetriggers the illumination of the background light source and a fallingedge of the single trigger pulse triggers the illumination of theforeground illuminated image.
 18. The imaging system of claim 16,wherein the foreground light source and the background light source areflash light sources that generate visible light.
 19. The imaging systemof claim 16, wherein the foreground illuminated image and the backgroundilluminated image are color images.
 20. The imaging system of claim 16,wherein the digital camera outputs the foreground illuminated image andthe background illuminated image as digital image files.